Repository navigation
Keep two timing tests of the command line green on a busy Windows runner - #433
Merged
Merged
Conversation
Two restest-cli tests failed on windows-latest on 30 October 2026 and passed when re-run, with nothing under src/main changed. The refused-lap test timed the whole run from its start against four seconds, two of which the slow 401 itself takes since it was raised from 0.8 s; a Windows runner took 4.155 s. It now times from the 401 to the run's end: a correct run ends a fraction of a second after it, one that keeps the slot it took ends five seconds after it. The rate test's run spent 4.5 s of its 2 s budget starting up and sent nothing. RESTest pays for starting up out of the budget on purpose, so a run that had no room for three turns between its first request and the end of its time is now run again with twice the time, up to 8 s; every check is made on a run that had room. The two other tests of the class, ended by their number of requests, get the 30 s ceiling the other such command-line tests use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: None.
What you can do now that you could not before
Nothing changes for someone running RESTest: only tests change, and nothing under
src/main.Continuous integration stops going red by chance on a busy Windows runner. On 30 October 2026 two
command-line tests failed on
windows-latestand passed when re-run, with no code changed: therefused-lap timing test on Java 25 (run 41073186888, attempt 1),
and the rate test on Java 26 (run 41080563445, attempt 1).
Neither now counts a slow start against the run, and neither checks less than it did.
Try it yourself
From a clean checkout, with only Java and git. The three rate tests and the refused-lap test pass,
and Maven prints nothing:
To see that the refused-lap test still catches what it is there for, break that behaviour: make the
run keep the slot it took when the 401 stops it. The test then fails, five seconds after the 401
rather than within a fraction of one:
How it works
The refused-lap test checks that a run whose opening lap the API refused with 401 stops as soon
as the last 401 arrives, and does not then wait five seconds (the wait for late answers) for an
answer nobody owes it. By design that 401 takes two seconds to arrive. The test timed the whole run
from its start and wanted less than four seconds, so a slow start and a slow machine had to fit in
the other two. It now times from the moment the 401 arrived. A correct run ends a fraction of a
second later; a broken one ends five seconds later; the four-second line sits between them with
almost four seconds to spare, whatever happened before the 401.
The rate test checks that a run kept to ten requests a second spaces them a turn apart, sends
nothing once its time is up, and says how much quiet the rate made. The "nothing once its time is
up" part needs the run's time to end it, and RESTest takes the time it spends reading the document
and starting up out of that same time, on purpose. On the failing runner, starting up took 4.5
seconds of a 2-second budget, so the run sent nothing. Now, if a run had too little time after
starting up (fewer than three turns between its first request and the end of its time), the test
runs it again with twice the time, up to 8 seconds. On an ordinary machine it runs once, in two
seconds, as before. On a busy one it takes longer instead of failing. Every check is made on a run
that had room, and none is looser.
The other two tests in that class end their runs after 200 requests, not when time runs out, but
they still had only two seconds, starting up included. The same slow start would have failed them.
They now get the 30-second ceiling that every other command-line test ended by its number of
requests already uses, and they still check that their 200 requests took less than two seconds.
What changed in the code
RunLoopTest: the stand-in engineApiEnginenow records every answer, by operation and by themonotonic clock (
everyAnswer).firstAnsweredSince(operation, since)finds the first answer ofone run when the engine serves several, as it does in this test's loop over two patiences.
a_slow_401_to_the_lap_stops_the_run_when_it_arrivesmeasures from the firstlistSheltersanswerof each run to the run's end. The bound itself (
PATIENT.minusSeconds(1), four seconds) is unchanged.KeepingToARateCommandTest:runtakes the budget.keepingToTheRate(budget)clears thestand-in's request log and the captured output, then runs.
turnsLeft(budget)reads the firststored request and the report's
timeline.startedAt.LONGEST(four timesBUDGET) caps theretries.
CEILING(30 s) is for the two tests ended by their number of requests.assertFasterThanTheRatenow compares with
BETWEEN.multipliedBy(20)(two seconds, as before) instead ofBUDGET,which those tests no longer use.
src/main, no settings, no documentation.Evidence
./mvnw -q verify -pl restest-cli -amis green locally onmacOS — CI: run 41086627777,
green on its first attempt, Java 21, 25 and 26 on Linux, macOS and Windows
the changed assertion against the fault it guards by hand: with the slot release removed from
RunLoop(thesedabove), the test fails at 5.01 s after the 401 against the 4 s bound.after it with none. The whole run took 2.30 s and 2.02 s.
150 ms, 300 ms and 600 ms. The last run had five turns of room, sent six requests, and passed
every check.
Decisions taken
git log -S):the test came in with "Stop a run after a certain number of 401" (Stop a run after a certain number of 401 #422). There, the slow answer
took 0.8 s and the four-second bound left 3.2 s of room. Then "Keep a run to a rate given in one
line" (Keep a run to a rate given in one line: at most so many requests a minute, evenly spaced #425) raised the slow answer to 2 s, because a Windows runner had finished the lap after the
401 came, and did not touch the bound. That halved the room to 2 s, and the failure here took 4.155 s.
Raising the bound again would bring it closer to the five seconds a broken run takes. Measuring
from the 401 takes the two-second answer and the start out of the measurement. The moment is the
stand-in engine's own record of when it answered, not the moment a listener heard of it, since
listeners hear later and on their own schedule.
RunCommandsets the deadline at the moment the command started plus the budget, on purpose.The stall was 4.5 s (the run's own summary says "0 requests to 0 operations in 4.5s"), so a fixed
budget that would have survived it would be 8 to 10 s, paid on every build on every platform.
Running again costs nothing on an ordinary machine. The trigger is the run's own record (nothing
sent, or fewer than three turns between its first request and the end of its time), never a
failed check. A run that had room and sent too few requests, sent them too close together or sent
one too late still fails. The cap of 8 s makes sure that a run that never gets going still ends.
budget / turn + 2, now using the budget the run actuallyhad. I considered measuring it from the first request, which is tighter. But that bound is
within a request of what a busy machine produces, and this change is about removing chance
failures, not adding a stricter check.
the same cause, the same 4.5 s stall would have failed them, and the fix is the one
ARunalready documents for runs ended by their number of requests.
Open questions
None.
masterAGENTS.mdholddocs/DESIGN.mdwithout explicit approval🤖 Generated with Claude Code